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^^EmfflX)IJOGICAL Pm)BLBiS IN 
lliE EVi^ATION OF IKXOV^TIOX 

Martin Trcw 

Soiirces of Innovation in Higher EdiKiation 

Currently there is considerable ferment in American higher educa- 
tion arising frcM vddespread discontent with present arrangements and 
practices. This dissatisfaction has its roots in a set of develop- 
ments in higher education and in the larger society idiich are dianging 
the character and functions of our colleges and universities and at 
the same time changing expectations of idiat they should be doing. 

These forces affect individual ;_..5titutioris in very different— 
or even opposite — \^ays. For exanple, the steady increase in the 
percentage of the population which attends college raises the aca- 
demic quality of entrants to the more selective institutions, while 
it brings to less selective institutions large numbers of students who 
are there, at least initially, because there is not much else for 
them to do. The presence of large numbers of relatively unmotivated 
students in colleges which have no strong academic traditions poses 
problems similar to those which gave rise to the transfoimation in cur- 
riculum and in teacher-student relations in our high schools earlier 
this century. They are the problems, in brief, of generating in the 
classroom the interest and motivation which one could no longer as- 
sume the student brought w/ith him. The concern for relating the 
curriculum to the lives and interest of the students, rather than to 



a traditional body of loiovrledge or tlie specialized interests of the 
academic disciplines, imderlies many of the current efforts to change 
the form and content of instruction, especially at the undergraduate 



and IntTcductQxy levels. 

There are othex' forces viiich are luaking our traditioiiaj. forms 



of education less and less satisfying, in the graduate and profes- 
sional schools as weijL as in ilie uiaiergraouatc lioerai arcs colleges. 
The rapid growth of knewjledge makes the traditional syllabus obsolete, 
while sijiHiltaneously weakening the traditional boundaries oi tne 
acadonic disciplines. Closely related are changes in professional 
education as an increasir«gly wide range of knowledge oecomes directly 



relevant to effective professional practice; the gro^virjg role of the 
social sciences in the education of physicians, lawyers, engineers, 
architects, and city planners is a case in point. 

Ivhatsver its sources, the ferment has led to a variety of new 
approaches to higher education. These include the sivesping innova 
tions in the organizational forms cf higher education, such as cne 
consortium of institutions in California at Claremont or in hew 
Bigland in the Connecticut valley; the single institutions which 
embody some distinctive organizational principle, such as Santa Cruz s 
collegiate stiucture; the varied means of approaching what used co 
be called "general education;" and the latest effort any one of us 
may irjake to create a nev/ course around a problem, or a cluster of 



disciplines, a nsf v.ay of using teaching: assistants, or tiie new 

technology of electronic instruction. 

There are many kinds and degrees of inncvation, and the problen.^ 
of assessment of these varied efforts obvious Iv differ. Though it 
is difficult, I vdll tr>' to say samething about educatjonaj innova- 
tions, regardless of how far-reaching in intent they are or of where 
initiative lies. My er.phasis will be less on the tedmical problems 
of evaluative research— the relative strengths of different modes of 
investigation or different strategies ol analysi;>--thar on vhcjiac 
teristics of the phenomenon being studied and assessed and on the 
social context in which they are onbeudew. bhiit forces give rise 
to an innovation? Ivhat are the criteria of its success? IVho cares 
whemer or how it is assessed? These- are problems for the researcher 
which often override the Icnocty difficulties of how to measure c-iange 

or the influence of a clique of friends. 

I would like to address myself to innovations in Uio cuiri'.ijiun* 

and in the modes of teaching and learning, ratn.er tlian to innovat,.ons 
in broader organizational forr.s, which I think involve a soir.ewliat 
different set of ’’methodological problems. 

Some Functions of Educational Innov ations 

Innovations in instruction in higher education arise most often 
out of some felt sense of the inadequacy of existing arraj^tements , 



and veiy often from sheer boredcra ^^ith vdiat one has been ICe 

are always tinkering with our courses or *^ith the curricuim, eien 
idien they are workijig reasonably welx. Af/,t while pzx^sals, whether 
for a nel^ college or for a nei%- course, are iisually justified as pro- 
msing seme in5)TOveBient on what is being dene, veiy often we know 
or strongly suspect that what is proposed recommends itself not so 
much on its prcmiise of betteiment as on the certainty of its being 
different. 

An innovation is a break mth routine and habit; it disrupts 
Uirreflective ways of thinking, feeling, and behaving; it requires a 
heightened measure of attention and interest in the matters at hand; 
it forces the participants, and especially the creators, to think in 
fresh ways about familiar subjects, to reconsider old assumptions. 
Above all innovation dispels, if only briefly, the fog of boredan 
that hovers over everything we do in our classrooms. Habit and 
routine are extremely useful in allowing us to do a great many neces- 
saiy things without having to think about them, while freeing our 
minds and energies for other more demanding matters. But when habit 
and routines begin to encrust educational structures and processes , 
life, thought, interest, and creative imagination go out of them and 
they become boring to us and to our students. I think we knaw in- 
tuitively that boredan is a greater enemy of education than ignorance 
or error or even stupidity, that as an enemy it is rivaled only by 
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dogmatic authority. .And if boredcsn is a chief enemy, innovations 
and change are our chief ^-eapon against it because they can break 
through routines and release fresh energy, hnagination, and inquiry. 

I am suggesting that innovations m education justify themselves 
by their intrinsic qualities almost without regard to their outcomes. 
Indeed, innot^ation goes on constantly. For the most part it does nor 
advertise 5_tself by name, often because the innovator doss not need 
additional resources and because he does not want to become entangled 
in tJie cumbersane machinery through which formal changes in the cur- 
riculum are made. 

"Whether advertised or not, it is nT 5 >ortant that innovation is 
cdiEPionly done for its own sake and only secondarily for its outcomes . 
Because tliat fact greatly reduces the relevance of systematic evalua- 
tion of innovation, it reduces the significance of the manifest 
fonctior^ of evaluation- to tell the innovator what he has achieved 
and how successfully— as compared with its chief latent function— to 
legitimize an innovation and contribute to its continuation and ex- 
tension. Innovations will be made- -with or without evaluations 
almost regardless of their nature simply because we enjoy making 
them. From this perspective, evaluation studies are aimed less at 
the innovator tiian at funding agencies or course ccmmittees which 
can support or limit the life or scope of the innovation. Such 
studies thus are typically directed at expensive innovations or those 



idiich have a broader iropact on other parts of an institution and 
thus involve persuading others that the innovation has value and 
should oe supported. Innovations that are inexpensive or are con- 
fined within one department or one course usually are not ..valuai.ed, 

tiiey arc just enacted. 

To emphasize the latent functions of same social pattern or 
practice liie evaluation is implicitly to minimize the significance 
of its manifest functions. A large part of evaluation in education 
is best understood as a form of persuasion directed at powerful 
people who make decisions and control resources. But this is not 
to say that evaluation studies need be nothing more than devices 
for legitimation and persuasion, hi modest ways evaluation can 
help shed light on educational practice, and perhaps help us see 
lAat an innovation actually consists of as well as what it achieves. 
But the context and function of such studies affect the way we con- 
duct the studies and how much confidence we can place in the findings, 

and thus are deeply inrolicated in metnodoj-u^y . 

I would therefore like to discuss three aspects of research and 

iimovation in higher edication. First, the political context and 
significance of evaluation. Second, the educational and research 
problems posed by the diffuseness of the intended outcranes of educa- 
tion, including its innovative forms and the long delay beyond the 
college years before many of these outcomes manifest themselves . 



Third, the great difficulty, especially in innovative courses, of 
distiriguishing the special circumstances sarrouijding their creation 
and adoption from the other characteristics of the innovation i^hich 
may reconsnend it to others and to its inscitutioiialiiation. 

The Political Context of Research in Higher Education 

We see in Merican higher education a grovdng sense Ox the 
relevance of systematic research procedures along ivitli a consider- 
able hostility to social research and suspicion of the educacional 
ii5)lications of its firidings. Paradoxically, both the giwing 
need for such research and the ivariness of it rise from similar 
sources. The rapid growth and democratization of higher education— 
a groivth idiidi brings into our colleges an enormous number and variet'/ 
of students whose values, motives, and purposes are strange to the 
academic man— leads to the extension of social research in many col- 
leges and universities . Moreover, conditions in the large public 
colleges and universities make it difficult to establish the old per- 
sonal relation of student and teacher, and thus lor the faculty 
member to know his anonymous students in any real sense. 

Increasingly, and often for much the same kind of practiceil 
reasons which prompted the social surveys of the 19th Century, educa- 
tors are turning to social science to learn the xacts about their 
students that are no longer directly knowable by the teacher or ad- 
ministrator. But this return is met v/ith the same ambivalence among 
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cultivated men as was the earlier developr.ent of social researdi. 

It threatens the academic man's role as an intellectual and as an 
interpreter of his own social experience; it asserts that mch that 
is of importance- -not only in the wider societ-y but in his own 
classroom and in the students' residence halls --no longer can be 
adequately loioim and uriderstood by the man of intelligence and sen- 
sibility, The suggestion— often made tactlessly by social scien- 
tists— that the professor of hiSianities cannot grasp the social 
processes going on around him without the aid of tl*e social scien- 
tist's special skills and techniques is frequently met v/ith hostility 
and resentment. The very existence of social research on campus, 
as sane professors have candidly stated, is an insult to their in- 
telligence. Their response, made perhaps with more feeling than 
logical consistency, is to doubt that social science is more than a 
pretentious fraud and, at the same time, to fear its manipulative 
consequences , 

But social research threatens not only the intellectual com- 
petence of academic men regarding their teaching functions; it is 
also felt by sane to be a threat to liberal education. Colleges where 
educational practices and arrangements are seen as <mibodying the 
rvalues of the institution, instead of merely facilitating their at- 
tainment, are likely to be inhospitable to the notion of applying 
the findings cf social research. To the extent that its practices 
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have become highly institutionalised, charged Kith value in then- 
selves, a college Kill resist conscious plannij^ based on rationaxi *-d 
procedures arid data. Such an institution is lihely to rely on cccmiutee 
deliberations as inore likely to preserve the primcy cf the substan- 
tial values. By contrast, a collie ccxnmitted to tlie achieveinont o£ 
easily measurable goals, and uhich is prepared to measure ai^ iriodify 
its practices against the criterion of the efficient achievement of 
these goals, is more likely to sponsor and apply social research 
aaainst ivhose findings elements of the organisation can be evaluated. 

Liberal education is a substantial value in itself. It is the 
practices and relationships and patterns of benavior that enter into 
it, much more than it is some nebulous "outccsiie," difficult if not 
iiT5)ossible to measure. By contrast, vocational and professional 
education is to a much greater extent instrumental and goal-oriented— 
the outcomes are measurable in skills and knovjledge acquired, examina- 
tions passed, diplomas earned. The colleges and the parts of large 
universities that are deeply committed to liberal education have been hss 
likely to welcome or apply social research which touches on their 
core values and acti\dties than have those organizations or parts of 
organizations whose practices are instrumental to s(mie more clearly 
defined or measurable goals. 

Typically, in American colleges and imiversities, power is dis- 
tributed in extremely complicated and obscure ways among ''he adminis- 
trators, the faculty, the trustees, and various important constituents. 




such as alunmi, the current body of parents, and — in tlie case of 
pi 61 ic institutioris — thr legislature or otiier sources of public 
funds. The question of uhat is nonipulable and by whan is itself 
highly uncertain, at least as difficult to know as patterns of stu- 
dent behai’^ior, vdiich inay the nooinai subject of investigation, 
ihrery organization is to some extent a polity, in which political 
processes deteimine who can initiate what events, who can veto 
then, and whose consent must be gained before policies are put into 
effect or sabotaged. Some studies of internal organizational pro- 
cesses have been done within foimally bureaucratic organizations 
(e.g., business firms) and within foimally dentjcratic organizations 
(e.g., trade unions and political parties), but almost nothing has 
been done by vay of studying the political processes witliin institu- 
tions of higher education. These institutions are in part bureau- 
cratic and in part democratic, combining the principles of hierarchy 
and colleagueship in varying degrees. I am not suggesting a design 
for the study of colleges and universities as political structures, 
but rather that the relevance of social science to educational policy 
cannot be discussed without recognizing that policy reccsTimendations 
within colleges are quickly transformed into political issues. 

A highly rationalistic conception of the relation of lesearch 
to policy obscures the political character of a college and of 

to it: those iho liold such conceptions are continually 
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surprised and iiidignant idien the institution does not take Uie 
"reasonable" course of action suggested by researdi- A director of 
the Bureau of Institutional Researdi at a large mid-i.estem univer- 
sity has described, with beccsning candor, actions taken by faculty 
coimittees in tico cases in which his bureau conducted research on 
the issue in question- -both actions were at variance mth the ap- 
parent indications of the research. He observeil, witli more sadness 
than anger, that "actions such as these represent one of the frustra- 
tions of a person in institutional research. The mere establishment 
of an institutional researdi unit does not in itself guarantee tliat 
decisions vail be made on a more realistic, objective, and reasonable 
basis. As you can see, even in our institution with its long tradi- 
tion of faculty- oriented institutional research, faculties and faculty 
ccssiittees have been known to make decisions on other than a purely 
objective basis." 

Without describing these cases in detail, I can report only that 
the research center’s recommendations are "realistic and objective” 
on the basis of a rather narrow conception of educational efficiency, 
and that faculty members with other values regarding education might 
vrell see sudi a research report as a uolitical document and oppose 
it as such. But the claim to objectivity denies the value implica- 
cations of the research and makes opposition to it mere pigheadedness, 
or in the words of this research man, "stubborn resistance to change.” 
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•fids in itself tends to eccite suspicion of all social research aniong 
fecalty nieabers viiose values are frequently at variance mth those 
hi5)licit in, but denied by, offices and bureaus of research-a point 

to vMch I TOuld like to return a little later. 

•Hie general principle that policy recoomendations (v.hether they 
are or are not based on social research) are in most cases n-nr.ediateiy 
transformed into political issues alerts us to a number of politically 
relevant factors intervening betiveen research cuid nsplenwntation. Th.. 
distribution of paver in colleges ana urdversities is more diffuse than 
in most formal organizations. Tne principle of bureaucracy tends to 
centralize formal paver and authority at the top of the hierarchy, »d.ile 
the principle of colleagueship tends to spread it more ividely among the 
feculty. There is sane evidence of the existence of a long-range trend 
toward the diffusion of power by means of a strengthening of the principle 
of colleagueship and of faculty participation in the government of col- 
leges and universities. The AAUP, for example, finds that over the 
past several decades faculty influence in most or the colleges they 
have been studying has been growing. There seems little doubt that 
this tendency is a result of the strong efforts American colleges 
and universities are making to upgrade themselves to the level of tlie 
more distinguished colleges and universities idiere the principle of 
colleagueship is most strongly and influentially established. One 
result of this tendency is for the interests and values of the faculty 



to become ii»re ividely and more directly involved in the application 
of social research to educational policy. Thir. in turn inarvcs it in 
creasingly difficult for admLnistrators to act ivith authority, even 
on the basis of research findings anc recommendations. 

The interests of the faoaity are touched at many points by pro- 
posals to modify the structure or content of an educational program. 
Clearly, areas of investigation vary in the degree to idiich they 
visibly mpinge upon the interests of tlie parties concerned. In 
general colleges ivlll be more receptive to applied research on issues 
further removed from the interests of those ivho wake the decisions— 
more hospitable, for example, to research on student life than on 
faculty authority, to research on the social implications of residence 
hall architecture than to studies of the distribution of pcv^er in 



college and university departments. 

In the United States the bulk of applied research in higher 
pa^ira i-^nti has been carried out by fact-finding agencies within the 
colleges and universities-by assistants to the president, fay deans 
or assistant deans, by testing offices, arid increasingly by oifices 
of institutional researdi. Hie line between social statistics and 
gpr-ial science is a fine one and lies in the shift of a passage in 
a report. .Absence levels, for example, may be indicators of under- 
lying social and institutional processes; the next step is to study 
these processes more directly. If this step is taken relatively 



rarely, it is partly because the people idio do this kind of research 
for colleges and universities rarely are social scientists or have 
an interest in organizational analysis, and partly because of the 
suspicion mth iliich research is viewed by important parts of the 
faculties of nany institutions. 

SoTiC of this suspicion has a different basis from that \diich 
arises from the dispute betiveen humanists and social scientists over 
the rel-^tive poi^^er of science and sensibility for interpreting social 
life, but its effects are similar and reinforcing. The suspicion 
arises frcmi a profound struggle that goes on within many institutions 
and takes many different forms, a struggle bet^icen tliose committsd to 
some ideal of liberal education — to the development of the intellec- 
tual powers of the individual, of his breadth of vision, independence 
of mind, and critical faculties --and those primarily interested in 
education for extrinsic ends, for social and vocational skills. The 
suspicion of research held by many humanists is that in this struggle, 
basically a political struggle over the means and ends of education, 
research is usually on the side of the vocationalists . 

It is thought to be so, not only by virtue of the kinds of people 
who do it but also by virtue of the V0iy lei. nds of c3ata they collect . 
For while the indicators of success of a liberal education are likely 
to be vague, difficult if not impossible to measure, and scarcely 
distinguishable from the effects of all the other experiences a 



student has had in his life, the indicators of successful training 
are the kind of performances that testing offices and offices of 
institutional research can measure. cognition of this by those 

faculy nieinhers ccmiriitted to liberal education, ano. the suspicion 
that arises from it, partly explains the mechardsms that surround 
offices of research to insulate them from the core values and actixa- 
ties of tlie faculty; for examole, their subordinate status and their 
definition by the institution as technical agencies gathering statis- 
tical infomation primarily for administrative uses, rather than for 
basic research into the rnture and processes of higher education. 

The criteria and indicators of ’’success” of educational prac- 
tices or innovations that are employed in educational research are 
elements in the academic-political controversies on many campuses. 
They affect the forms that research takes and the reception it gets-- 
that is what happens to it. 

Proximate and Ultimate Goals of Education 

Some ’’outcomes" of education are easily measured, and for that 
reason, as well as others, they are commonly measured. Among these 
are the student's grade point average, drop-out or transfer rates, 
achievement of graduate scnolar ships and higher degrees. These 
matters are part of almost every research into nigner 




not only because they lend themselves to easy and systematic measure- 
ment, but also because they are important in th^nselves. Grades are 
not merely an "index’* (hov:ever v:eak) of what has been learned; they are 
also an important determinant of the individual's future opportunities 
2nd life chances, among them his chances of gaining admission to a 
good graduate school. Acceptance by a good graduate school is an even 



more nnportant determinant of a man's chances of making significant 
contributions to science or scholarship. 

But whatever their objective importance, which is very great, 
grades and higher degrees are inadequate measures of the outcomes of 
educational experience for many reasons. Hiey do not measure the 
whole of what some men wush education to do to or for students. They 
are poor measures, for example, of the success of a liberal education 
in refining serisibilities , developing capacities for critical and in- 
dependent thought, or the use of reason and evidence in everyday life, 
or the enhancement of the individual’s capacities for enjoying life 
and making fruitful contributions to it. Some men w-ant these great 
goods to flow from a scientific and technical education as well. The 
difficulties in discovering idiether indeed an education has these 
effects are several. 



In large part, these qualities of mind and spirit do not shc«A^ 
themselves, during the college years, but m.ay be laid down then as 
potentialities which bear fruit in later life and career. They are. 



for the most part, exceedingly difficult to measure systematically, 
hcRvever lUicli vire pride oursel\’es on our ability to recognize their 
presence or absence in others > Moreover , tnese Qualities are not 
only valued outcomes of formal education, but also the products of 
the iviiole of man*s genetic equipment and life experience. Even if 
vre could measure them i\d.th some precision and confidence, hoiv are ive 
to distinguish the part played by formal higher education from all 
the other more enduring and onotionally v;eightier influences on a 
iriaTi'S life and character? 

In a word, then, the most important and truly valued outcomes 
of higher education are extronely difficult, if not impossible, to 
assess. As a result, many institutions, usually those witli the 
least firm educational purposes and the least distinctive character, 
fall back in their self-assessments on those presumed outcomes of 
higher education that are most easily measurable. And, in a familiar 
translation of necessity into virtue, such an institution may define 
its aims in terms of idiat can be measured, and to shape and justify 
its practice in terms of its success in reducing the drop-out rate, 
increasing the number of fellowships its graduates earn, and the 
like. 

TVhat are the alternatives for the institution that does not v/ant 
to reduce its educatior^al aims to the level of the most easily mea- 
sured of student characteristics? Matters are not quite so hopeless 



as my remarlcs above luay seem to suggest. There are things that 
research can do to help an institution assess its success in achiev- 
ing its most profound and not merely its most pioximate aims. For 
example : 

1. We are not confined to the study of the most obvious and 
easily measurable outcomes of education- There are v;ays to explore 
changes in basic values and attitudes of students, and even aspects 
of their personalities ivhich education aims to modify over the col- 
lege years; to explore changes in life plans and the conditions and 
experiences in the institution which give rise to them; to attempt 
at least to study such subtle matters as creativity and independence 
of mind and judgment. 

2. These are all to a considerable degree a product of the 
student *s life experience before coming to the institution. To some 
degree we can assess the extent to which they are already present at 
entrance, so that we can make some assessment of the relative efficacy 
of different educational practices during the college years in develop- 
ing (or inhibiting) these qualities. 

3. We can do far more than has been done to follow our graduates 
into their adult careers to see vdiat happens to them there, and to 
see if we can make even tentative inferences about connections betvveen 
their adult careers and theii college experience. 



The Influence of Experimental Research on 
Educatioiial Practice and Its (Xitcomes 



Different forms of social investigation vaiy in the extent to 
idiich they affect the educational processes that they aim to illumi- 
nate. A survey of a collegers alumni presumably would have little 
direct- influence on the faculty and students at the institute at 
the present time, (^estionnaires distributed to entering freshmen 
probably mil have relatively little effect on their subsequent be- 
havior, though repeated questioning about a given issue— say, the 
question of student-f acuity relations --might be expected to increase 
the salience of that issue in the minds oi the siudents . But experi 
mental research- linked changes in the curriculum are likely to have 
very marked consequences for the teaching- learning process oyer an^ 
above those effects which the alteratioris are specifically intende d 
to achieve. It may be worthwhile to consider for a moment the problems 
such experiments pose for research designed to assess their effects 
and effectiveness. 

First, there are the dif f iculties , already discussed, of measur 
ing the genuinely desired outcomes and of disentangling them fo'cxn the 
manifold extraneous influences of life and time outside the experi- 
mental classroom. Experiments share this difficulty, as I have sug- 
gested > with other forms of research into education. 
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In addition, expericents in education, like social experironts 
in general, pose special difficulties for research, in that they 
introduce into social situations poicer^l forces over and above those 
purposefully introduced by the experiment. These 'other forces 
affect the outcor.es of the experiment in ways that are very difficult 



to separate from the effects of the "intended" experimental pro- 
cedures, that they so closely resemble then. The general phenomenon 
to ivMdi I am referring has become knoivn as the "Hav-rtnome effect, 
after the famous experiment on icorker productivity at the Hawtiiome 
plant of the Western Electric Ccmpany in the late 1920 s. That 
study shrr^^ed that the experimental situation itself, independent of 
the purposeful manipixLation of the situation, modified social rela- 
tions, group morale, and individual motivations among the subjects 
in ways that affected their perfoimance, in most cases for the better. 

This phenomenon has beccmie widely associated witli the independent 
and common observation that in education no experiments fail, so that 



it has been seriously suggested that one educational strategy would 
be tc "institutionalize the P^^rthoxne effect" by making "experimental" 
innovations a regular part of school or college administration. This 
advice has not been widely adopted because institutions are made as 



unhappy as individuals are by a steady diet of innovation; it puts 
a strain on lines of communication and authority, makes the coordina- 
tion of the different parts of the institution more difficult, and 




rakes life less predictable and thus riore unsettling and anxiety- 
arousing for the individual. Hie gains of educational innovations 
nsy be Kortii ail this, but before recocnrending such a strategy which 
dissolves the distinction betireen "action" and '*research" by iraking 
the research itself the action, it riay be v:orth considering what are 
the forces involved in such "experiments" to see if indeed they can 
be made part of the institution* s regular procedures without their 
unsettling side effects. Put another way, what are the sources of 
their evident poiver to raise performance? 

1. One of the forces generated by a classroom experiment is to 
make the **subject" students feel somehovj distinctive, a ''special" 
groLQ) getting special attention. This effect of the experimental 
situation was noted at Haw^ome, where it presumably generated among 
workers there the special groip morale and commitment to the task 
that resulted in their higher individual performances. 

2. Quite distinct from that process, however, is tlie fact that 
experimental courses are customarily instituted and taught by imagina- 
tive teachers, who have given an extra measure of thought and effort 
to the pedagogical problems they face — the innovators themsei ves , one 
may guess, are probably better than average teadiers. This cannot 
help but play a part, perhaps the major part, in their customary 
"success." 
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3 . Sot only is the self-selected staff of an expermintal clas= 
likely to be note gifted than the average but they are also litely 
to have a strong interest in the success of Aeir "experuBent,” and 
to co«unicate that interest through the enthusiasm mtb -.Aich they 
tackle the course. Bithusiasn for a siiject is a Kell-kno«n charac- 
teristic of the successful teacher, even in more routine courses. 
Coupled with the innavative character of an "experiment,” it is a 
pcKverful pedagogical force. 

4. Typically, <if not uniformly, "experimental” courses have been 
assigned larger amounts of the institution's resources than have com- 
parable "routine" courses. The ratio of teachers to students is 
higher, and the amount and intensity of student-teacher interaction 
is commonly greater in "experimental” than in routine courses. This 
also helps educational "experiments" to succeed, both through the 
more thorough way in which the course material can be covered with 
each student and through the hi^er levels of student motivation that 

teacher attention can generate. 

liudi of the success of an "experimental" course is related to 

the fact that it is a break in routine which forces a higher level 
of imagination and energy frcrni the staff and excites it in the stu- 
dent. Tlie sheer innovative character of such an "expernnent," coupled 
mth its typically rich endowment of resources by the institution, 
almost ensures its success independent of its purposeful content. 




But the problem for research vMch aims at asssssirg the Aorth of 
an educational innovation is clear- hCiif to distinguisii the expci- 
mental effects from the designed or purposeful effects. It ray be 
aroued that the time to assess an innovation is idien it is no longer 
an innovation, when it has becane routinized and no longer can call 
forth the special energies, resources and enthusiasms of an "experi- 
ment." The trouble is that an institution usually wants aii assess- 
ment of an experiment in Hie curriculum before it has committed major 
resources, before ii; has made the necessary organizational adjustments, 
and before it has perst^ed people who did not initiate it to staf a. 

it. 

I havre Qiiphasized the difficulties for research In assessing the 
worth of a curricultan experiment, but I do not waiit to exaggerate 
them. Research methods of several Rinds can be enployed to explore 
the workings and outcomes of an experimental course, and such research 
may be of real value to the institution so long as the policy-makers 
recognize the special characteristics of educational experiments that 
make them so difficult to assess. For one thing, the degree of "suc- 
cess" of such a course, whatever its sources, can be tested at its 
conclusion by using the ordinary indicators of comparative perfor- 
mance on examinations, or more subtle indicators of intellectual 
powers and creativily that might be devised. Another approach is uo 
try to identidfy the pedagogical forces set loose by an innovation by 



to close am ccntimcus observa- 



subjecting the ejqserir-ental course 
tion- aimir^ to see *^hat elements in it call forth ghe greater rastiva- 
tien and effort that I assume will be obsen^ed. Such observation, 
of course, should also be accompanied by parallel observation in 
"ordinary” classes covering the same or comparable materials, to 
alloiv something approaching a comparative analysis of the observa- 
tional data. It may well be that such observation will alloiv the 
research to identify aspects of the cairse— pedagogical de\i.ces, 
organization of the subject, or whatever— which, though not explicitly 
"intended" by the innovators, appear to be partiailarly successful, 
and which might he more wddely introduced into the oirriculum on a 
regular basis. In a sense this would be an effort to separaue the 
pedagogical forces associated mth innovation from irinovation itseoj.. 
It would he an attempt not to institutionalize innovation but rather 
to identid^ those of its elements which are not dependent on the 
presence of the innovator or extra resources. Knowledge of the 
genuinely effective aspects of educational practice might liberate 
institutions from reliance on the specific educational forms in 
which they manifest themselves , allowing the invention of new forms 
which embody the effective processes in more effective or less ex- 
pensive ways. To my loicwiedge, this kind of obs8j.vation has not 
often been done on a s}’’stematic basis in educational institutions, 
and while the value of such observations is heavily conditioned by the 
skill and sensitivity of the observer, it very much warrants trial. 



Illumiimtive versus Evaluative Research 



I have been discussing thus far some of the problems of evaluative 
research in hi^er education: difficulties rooted in the suspicions 

of humanists and the conflicts idthin faculties; difficulties in tlie 
criteria ve use to assess educational efforts and in the re^ioteness 
of ultimate goals from proximate outcomes; difficulties in disentang- 
ling the unique qualities of innovative teaching procedures from their 
enduring and transferable qualities. But I feel an obligation here 
to end on a more hopeful note: to suggest that these difficulties 

are superable, that they are Korth the effort needed to deal vjith 



them. 

*2^0 fixst issue has to do ii?ith the institutional context of 
evaluation: vho does the job, to vhcM does he report his xindings, 

and idiat is done ivith his report? Insofar as evaluation is done by 
a research aim of the administrator, xdio reports to the administration 
regarding the value of certain aspects of the curriculum, the research 
enterprise is likely to face considerable suspicion and hostility 
from the faculty. As I have suggested, much o^ che suc^picion is 
merited, since evaluation must be predicated on educational values, 
hoi^ever disguised as science, anci these values are very ofuen I mighi- 
almost say chronically- -m dispute. The iv'ay for e/aluative research 
to meet this saspicion involves tico changes in the character of such 
research. First, research on innovative efforts must be seen from 



the beginning as "iliumtnative" rather than as "evaluative*' in the 
narrow sense. It imist recognize that the value of innovation also 
cau5)rises the rewards gained by the faculty ineofibers vsiio create it 
and are not confined to its easily measured outcomes. Second, 
researchers must recognize that these outcGnes bear only a remote 
relation to the ultimate impact the faculty monber may be hoping to 
Imve on tiie minds, characters, and lives of his students. Ibis 
means that such researchers must forego the dubious pleasure of 
awarding gold stars and demerits to acadonic innovators, but must 
tiy instead to serve thsn. Research on iimovatiGn can be enlighten" 
ing to the innovator and to the whole acadonic comiriunity to whom 
research reports are properly addressed by clarif>»^ing the processes 
of education and by helping the innovator and interested other parties 
to identiJfy those procedures, those elements in the educational ef- 
fort, which seon to have had desirable results. Such research may 
involve a comparison of proximate results, such as examinations, 
papers, and so forth, with those produced by other more conventional 
courses. It may also involve close semi -participant observation of 
the course in an effort to identify the operative social and psycho- 
logical mechanisms vhich the innovative procedures create (often 
beyond anyone's intention) vhich engage the interests and efforts of 
students and open them to the instructor's attempts to transmit 
skills, broaden horizons, or deepen understanding. Precise techniques 



of iiKiui.Ty are not at issue here; v.ie knov; pretty ivell their charac- 
teristic strengths and limitations- Vdiat is important is tliat tne 
research he seen to he in the service of the innovative enterprise, 
and not sitting in judgment on it. And for that, it must accept its 
CMii tentativeness and function as a facility of the fs^cxxlty , not as 
a part of the administrative apparatus. The formal status of the 
researcher or the research group, vdio einploys him, to i-dicxTi he 
addresses his findings, and hoiv’ he avoids being dra\in into academic 
controversies , are crucial here, though circumstances differ enough 
that no set of recommendations on these matters can possible apply 
to ail iristitutions . 

In considering the gulf hetiireeri proximate indicators of the 
results of educational innovation and their long range goals, wisdom 
resides, I believe, in a decent regard for the limits of research, 
hhat are the qualities that make creative engineers, resourceful 
businessmen, thoughtful and responsible citizens, men of independent 
mind, mora:* sensitivity, and aesthetic sensibility? hbat relation 
does college performance bear to these qualities? And what influence 
do specific educational arrangements have on idiat men do and what 
they are in their lives? A consideration of the kinds of men vh.o have 
been exposed to the most varied kinds of higher education, or to none 
at all, should make us pause before we give any ready answer. 

hhen we return, as does the teacher himself, to the student be- 
fore him, we may attend to ivhat we see not merely as a most imperfect 



indicator of futjxre achiev&T.enc of qualities, but as of intrinsic 
importance. On one hand it is important that students be able to 
learn and be able to demonstrate that they have learned assigned 
material— important for its effect on the range of possibilities 
that are open to the successful students but closed to the academic 
failures. On the other hand it is at least as important vdiether 
students are bored or engaged, committing their energies or cooly 
mthliolding them, fulfilling obligations or freely involving them- 
selves in learning, imd these things as ve know can be affected by 
educational arrangements and procedures --however constrained uy 
deeply set qualities of mind and character that the student brings 
with him to college and that remain with him unaffected there. We 



also know with Woodrow^ Wilson that: 

The real intellectual life of a body of under- 
graduates, if there he any, manifests itself, 
not in the classroom, but in what they do and 
talk of and set before themselves as their 
favorite objects beti\^een classes and lectures. 
You will see the true life of a college --v/here 
youths get together imd let themselves go upon 
their favorite themes’>--in the effect their 
studies have upon them ^dien no ccmpulsion of 
3uy kind is on them and they are not thinKing 
to be called to a reckoning of what they know. 



We know also that the life of the student outside of class can be 
influenced by our efforts. The innovator can see seme of this, but 
he is busy teaching. The researcher can see more, much more. He is 
trained to see just those things, and he is less constrained to see 



-i^iat he hopes to see. The illumination of educational innovation 
through systematic research can be in large part the identification 
of those educational processes that can be linked to the innovation- - 
the processes of learning and groivth that go on inside and outside 
the innovative classroom, laboratory, or residen^p' hall. 

Finally, with regard to the uniqueness of innovation and the 
special resources of talent and imagination frequently available, 
it may be that research should attend precisely to those qualities 
of abundance, rather than trying to '*partial them out” in assessing 
the effects . It may be that what we should aim for is not so much 
the routinization and institutionalization of successful experiments 
but a climate and organizational arrangements vhich make innovation 
easy and frequent. If, as I suggested at the outset, innovations 
recommend themselves for their intrinsic qualities rather than for 
their putative outcomes, if they are our chief weapon against bore- 
dom and routine, then the real research effort should be directed 
toward the effort to "evaluate” them once in being, 'fhis posture 
is completely compatible with the aim of illuminating their processes 
and proximate gains. We can w^ant to encourage innovation, while 
recognizing that some experimental efforts will be more successful 
than others, judged by their ovai and by broadly accepted criteria. 

We need not set aside all "academic standards" or notions of crafts- 
manship and achievement in a unreflective celebration of 
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spontaneity. 111616 is enough ai ti-ii.tellectualism afloat 
today, both inside and outside the academy, without social research 
contributing any more. But here we ccme very near to hhat is perhaps 
the central problem for the student of educational innovation. For in 
innovation we are very often dealing with "enthusiasm” on the part of 
innovators and sometimes of their students as well. On one hand, this 
enthusiasm means heightened attention, alertness, involvement, ccsnmit 
ment, creativity; on the other, the danger of enthusiasm lies in the 
passion of the true believer and of his terrible certainties. If our 
studies of educational innovations can illuminate those forces vdiich 
are respectively the chief instruments and enemies of education, we 
can, perform a very considerable service to our students and to our 
innovative colleagues, and to the enterprise of learning. 



